Handle multiple providers/llms #45

mbertrand · 2025-01-31T21:48:55Z

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/6622

Description (What does it do?)

Adds the langchain-community library and uses the ChatLiteLLM class for all LLM models.
Updates the LiteLLMProxy default kwargs to work with this class.
Adds support for one bedrock anthropic model in the LiteLLM proxy server configuration.
Sets different configurable default models for each bot.

How can this be tested?

You'll need this env value set (same as heroku mit-learn RC) in backend.local.env : OPENAI_API_KEY
Start the containers and go to http://ai.open.odl.local:8003. Ask some questions, it should work. Look at the logs, you should find this:
```
web-1         | LiteLLM completion() model= gpt-4o; provider = openai
```
Add AI_DEFAULT_RECOMMENDATION_MODEL=openai/o3-mini to your backend.local.env file and restart containers.

Try the chat UI again. It should still work and you should see this in the logs:

web-1         | LiteLLM completion() model= o3-mini; provider = openai

Add the following env settings to backend.local.env:

AI_PROXY_URL=http://litellm:4000
AI_PROXY_AUTH_TOKEN=sk-secret
LITELLM_MASTER_KEY=sk-secret
LITELLM_PROXY_API_KEY
AI_PROXY_CLASS=LiteLLMProxy

Restart containers and interact with the chatbot again. It should still work and logs should indicate that the requests are going to the LiteLLM proxy service:
```
  litellm-1     | LiteLLM completion() model= o3-mini; provider = openai
```

mbertrand · 2025-01-31T21:49:52Z

env/backend.local.example.env

+# AWS_ACCESS_KEY_ID=
+# AWS_SECRET_ACCESS_KEY=


Required for AWS Bedrock models

mbertrand · 2025-01-31T21:50:24Z

fixtures/common.py

+            "ignore",
+            module=".*(pydantic).*",
+            category=UserWarning,
+        )


Pydantic warnings coming from ChatLiteLLM class

mbertrand · 2025-01-31T21:51:40Z

ai_chatbots/chatbots.py

            **(self.proxy.get_api_kwargs() if self.proxy else {}),
            **(self.proxy.get_additional_kwargs(self) if self.proxy else {}),
            **kwargs,
        )
-        if self.temperature:
+        if self.temperature and self.model not in settings.AI_UNSUPPORTED_TEMP_MODELS:


The new o3-mini model does not support the temperature parameter and will raise an exception if it is passed along

mbertrand · 2025-02-03T11:34:16Z

ai_chatbots/chatbots.py

-            agent_graph.add_conditional_edges(
-                agent_node,
-                continue_on_tool_call,
-                {
-                    # If tool requested then we call the tool node.
-                    CONTINUE: tools_node,
-                    # Otherwise finish.
-                    END: END,
-                },
-            )
+            agent_graph.add_conditional_edges(agent_node, tools_condition)


This example was probably more verbose than it needed to be, the built in tools_condition function should suffice for most use cases.

mbertrand · 2025-02-03T11:37:55Z

ai_chatbots/chatbots.py

@@ -447,7 +421,7 @@ def __init__(  # noqa: PLR0913
        super().__init__(
            user_id,
            name=name,
-            model=model or settings.AI_MODEL,
+            model=model or settings.AI_DEFAULT_SYLLABUS_MODEL,


After AWS Bedrock access is set up, the default syllabus model should be changed to bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0

abeglova

👍

mbertrand added 2 commits January 31, 2025 14:56

Handle multiple providers/llms

192cbf0

Different default llms for different chatbots

a0536b7

mbertrand commented Jan 31, 2025

View reviewed changes

mbertrand added the Needs Review label Jan 31, 2025

some minor tweaks

2125d80

mbertrand commented Feb 3, 2025

View reviewed changes

mbertrand added 3 commits February 3, 2025 08:01

another unit test

cb82c6a

Tweak backend.env files

3eee686

remove unused constants

5936998

abeglova self-assigned this Feb 4, 2025

abeglova approved these changes Feb 4, 2025

View reviewed changes

mbertrand merged commit 9364145 into main Feb 5, 2025
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle multiple providers/llms #45

Handle multiple providers/llms #45

mbertrand commented Jan 31, 2025 •

edited

Loading

mbertrand Jan 31, 2025

mbertrand Jan 31, 2025

mbertrand Jan 31, 2025

mbertrand Feb 3, 2025

mbertrand Feb 3, 2025

abeglova left a comment

Handle multiple providers/llms #45

Handle multiple providers/llms #45

Conversation

mbertrand commented Jan 31, 2025 • edited Loading

What are the relevant tickets?

Description (What does it do?)

How can this be tested?

mbertrand Jan 31, 2025

Choose a reason for hiding this comment

mbertrand Jan 31, 2025

Choose a reason for hiding this comment

mbertrand Jan 31, 2025

Choose a reason for hiding this comment

mbertrand Feb 3, 2025

Choose a reason for hiding this comment

mbertrand Feb 3, 2025

Choose a reason for hiding this comment

abeglova left a comment

Choose a reason for hiding this comment

mbertrand commented Jan 31, 2025 •

edited

Loading